-
Notifications
You must be signed in to change notification settings - Fork 1.9k
opentelemetry: remove enforcement of timestamp fields for logs #10813
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Added logic to use the current host time when both timeUnixNano and observedTimeUnixNano are absent, removing the previous hard requirement for a timestamp in OTEL log records. Updated to prioritize observed_time_unix_nano when time_unix_nano is missing and to omit observed_timestamp metadata when no value is available. Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
WalkthroughAdds observed_time_unix_nano as a valid timestamp source for OTEL logs, falls back to the host current time when both timestamp fields are missing, and emits observed_timestamp metadata only when non-zero. Tests and README updated to reflect acceptance of records without timestamps. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor Client as OTLP Sender
participant Parser as OTEL Logs Parser
participant Encoder as Log Event Encoder
participant Clock as System Clock
Client->>Parser: Send LogRecord (time_unix_nano?, observed_time_unix_nano?)
alt time_unix_nano > 0
Parser->>Encoder: set timestamp = time_unix_nano
else observed_time_unix_nano > 0
Parser->>Encoder: set timestamp = observed_time_unix_nano
else no timestamps
Parser->>Clock: flb_time_get()
Clock-->>Parser: current time
Parser->>Encoder: set timestamp = current time
end
note right of Encoder: observed_timestamp included only if observed_time_unix_nano != 0
Encoder-->>Client: Encoded log event
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Assessment against linked issues
Suggested reviewers
Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro 💡 Knowledge Base configuration:
You can enable these sources in your CodeRabbit configuration. 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (29)
✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
CodeRabbit Configuration File (
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
src/opentelemetry/flb_opentelemetry_logs.c (2)
210-218: Bug: wrong error code returned for invalid spanId hex character.On non-hex spanId chars you return UNEXPECTED_TIMESTAMP_TYPE instead of INVALID_SPAN_ID.
- return -FLB_OTEL_LOGS_ERR_UNEXPECTED_TIMESTAMP_TYPE; + return -FLB_OTEL_LOGS_ERR_INVALID_SPAN_ID;
119-120: Fix tv_nsec computation in flb_time_from_uint64
The current implementation subtracts seconds from the full nanosecond count—yielding tv_nsec values far exceeding 1e9 (e.g., 1640995198359004800 for input 1640995200000000000). Replace with modulo to correctly compute sub‐second nanoseconds:--- a/include/fluent-bit/flb_time.h +++ b/include/fluent-bit/flb_time.h @@ static inline void flb_time_from_uint64(struct flb_time *dst, uint64_t value) - dst->tm.tv_sec = (long) (value / 1000000000L); - dst->tm.tv_nsec = (long) (value - dst->tm.tv_sec); + dst->tm.tv_sec = (long) (value / 1000000000ULL); + dst->tm.tv_nsec = (long) (value % 1000000000ULL);
🧹 Nitpick comments (4)
tests/internal/data/opentelemetry/test_cases.json (1)
40-46: Align “missing_timestamp” expectation with current JSON group-body shape.JSON path doesn’t emit an empty scope when no scope object is present, while other tests (e.g., “valid_basic_log”) also omit it. Recommend removing the empty scope here for consistency.
"missing_timestamp": { "input": {"resourceLogs": [{"scopeLogs": [{"logRecords": [{"body": {"stringValue": "test"}}]}]}]}, "expected": { "group_metadata": {"schema":"otlp","resource_id":0,"scope_id":0}, - "group_body": {"resource":{},"scope":{}}, + "group_body": {"resource":{}}, "log_metadata": {"otlp":{}}, "log_body": {"log": "test"} } },src/opentelemetry/flb_opentelemetry_logs.c (3)
248-274: Emit observed_timestamp when observedTimeUnixNano is an integer too.Currently metadata emission only handles string type. Accept positive integers as well (spec-compliant and consistent with time resolution above).
- if (observed_time_unix_nano != NULL && observed_time_unix_nano->type == MSGPACK_OBJECT_STR) { + if (observed_time_unix_nano != NULL) { + if (observed_time_unix_nano->type == MSGPACK_OBJECT_STR) { memset(timestamp_str, 0, sizeof(timestamp_str)); if (observed_time_unix_nano->via.str.size < sizeof(timestamp_str)) { strncpy(timestamp_str, observed_time_unix_nano->via.str.ptr, observed_time_unix_nano->via.str.size); } else { strncpy(timestamp_str, observed_time_unix_nano->via.str.ptr, sizeof(timestamp_str) - 1); } - for (i = 0; i < strlen(timestamp_str); i++) { + size_t ts_len = strlen(timestamp_str); + for (i = 0; i < ts_len; i++) { if (!isdigit((unsigned char) timestamp_str[i])) { timestamp_str[0] = '\0'; break; } } if (strlen(timestamp_str) > 0) { timestamp_uint64 = strtoull(timestamp_str, NULL, 10); flb_log_event_encoder_append_metadata_values(encoder, FLB_LOG_EVENT_STRING_VALUE("observed_timestamp", 18), FLB_LOG_EVENT_INT64_VALUE(timestamp_uint64)); } - } + } + else if (observed_time_unix_nano->type == MSGPACK_OBJECT_POSITIVE_INTEGER) { + flb_log_event_encoder_append_metadata_values(encoder, + FLB_LOG_EVENT_STRING_VALUE("observed_timestamp", 18), + FLB_LOG_EVENT_INT64_VALUE(observed_time_unix_nano->via.u64)); + } + }
100-111: Micro: avoid repeated strlen() in loops.Cache the length once to keep O(n).
- for (i = 0; i < strlen(timestamp_str); i++) { + size_t ts_len = strlen(timestamp_str); + for (i = 0; i < ts_len; i++) {- for (i = 0; i < strlen(timestamp_str); i++) { + size_t ts_len = strlen(timestamp_str); + for (i = 0; i < ts_len; i++) {Also applies to: 260-266
242-244: Use the function parameters for metadata key instead of hardcoding.Preserves flexibility and aligns with the function signature.
- flb_log_event_encoder_append_metadata_values(encoder, - FLB_LOG_EVENT_CSTRING_VALUE(FLB_OTEL_LOGS_METADATA_KEY)); + flb_log_event_encoder_append_metadata_values( + encoder, + FLB_LOG_EVENT_STRING_VALUE(logs_metadata_key, logs_metadata_key_len));
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (4)
plugins/in_opentelemetry/opentelemetry_logs.c(3 hunks)src/opentelemetry/flb_opentelemetry_logs.c(2 hunks)tests/internal/data/opentelemetry/README.md(1 hunks)tests/internal/data/opentelemetry/test_cases.json(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
plugins/in_opentelemetry/opentelemetry_logs.c (3)
src/flb_mp.c (1)
flb_mp_map_header_append(345-349)include/fluent-bit/flb_time.h (1)
flb_time_from_uint64(87-91)src/flb_log_event_encoder.c (1)
flb_log_event_encoder_set_timestamp(276-287)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (29)
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit, x64, x64-windows-static, 3.31.6)
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit (Arm64), amd64_arm64, -DCMAKE_SYSTEM_NAME=Windows -DCMA...
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 32bit, x86, x86-windows-static, 3.31.6)
- GitHub Check: PR - fuzzing test
- GitHub Check: run-ubuntu-unit-tests (-DFLB_ARROW=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, gcc, g++)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-24.04, clang-14)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, clang, clang++)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-22.04, clang-12)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-24.04, clang-14)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, clang, clang++)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-22.04, clang-12)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_COVERAGE=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, clang, clang++)
- GitHub Check: pr-compile-centos-7
🔇 Additional comments (6)
tests/internal/data/opentelemetry/README.md (1)
139-140: Doc update matches new behavior (no hard requirement on timestamp).The error code example change looks correct.
src/opentelemetry/flb_opentelemetry_logs.c (2)
24-25: Correct include.Including flb_time.h is needed for flb_time_get usage.
77-81: Good: fallback to host time when both timestamps are absent.Matches PR intent and prevents unnecessary drops.
plugins/in_opentelemetry/opentelemetry_logs.c (3)
24-25: Correct include.Needed for explicit flb_time_get usage.
221-226: Observed timestamp metadata is now conditional.Good: emits only when present (non-zero).
520-527: Timestamp resolution order is correct (time → observed_time → now).Matches OTLP intent and PR goal.
…ndling Signed-off-by: Eduardo Silva <[email protected]>
341ba03 to
4387ff0
Compare
Fixes #10793
Added logic to use the current host time when both timeUnixNano and observedTimeUnixNano are absent, removing the previous hard requirement for a timestamp in OTEL log records.
Updated to prioritize observed_time_unix_nano when time_unix_nano is missing and to omit observed_timestamp metadata when no value is available.
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.
Summary by CodeRabbit